<html>
<head><meta charset="utf-8"><title>source-based-code-coverage · general · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/index.html">general</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/source-based-code-coverage.html">source-based-code-coverage</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="216404359"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/source-based-code-coverage/near/216404359" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eh2406 <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/source-based-code-coverage.html#216404359">(Nov 11 2020 at 22:05)</a>:</h4>
<p>At <span class="user-mention" data-user-id="120518">@Eric Huss</span> suggestion I was going to try using <a href="https://doc.rust-lang.org/nightly/unstable-book/compiler-flags/source-based-code-coverage.html">source-based-code-coverage</a>. It looks like that option is provided on the nightly channel. So that is a relief did not want to build rustc. Is the <code>llvm-cov</code> tool also in the distribution? If I need to get it somewhere, how do I get it for windows?</p>



<a name="216404505"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/source-based-code-coverage/near/216404505" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eric Huss <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/source-based-code-coverage.html#216404505">(Nov 11 2020 at 22:07)</a>:</h4>
<p>I have LLVM installed through my package manager.  It looks like there is a PR opened a few hours ago (<a href="https://github.com/rust-lang/rust/pull/78947">https://github.com/rust-lang/rust/pull/78947</a>) to add the necessary tools to the llvm-tools rustup component.</p>



<a name="216404602"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/source-based-code-coverage/near/216404602" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eric Huss <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/source-based-code-coverage.html#216404602">(Nov 11 2020 at 22:08)</a>:</h4>
<p>IIRC, it is not too hard to install LLVM on Windows, I just downloaded it from their website.</p>



<a name="216405053"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/source-based-code-coverage/near/216405053" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eh2406 <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/source-based-code-coverage.html#216405053">(Nov 11 2020 at 22:13)</a>:</h4>
<p>Wonderfull! I will try again when that has landed. Or install LLVM if I get impatient.</p>



<a name="216680677"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/source-based-code-coverage/near/216680677" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Noah Lev <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/source-based-code-coverage.html#216680677">(Nov 13 2020 at 22:09)</a>:</h4>
<p>Not sure if this helps, but I've used grcov for code cov. It seems like it still needs some work though (it's not super accurate and for some reason I have to <code>cargo clean</code> every time I use it).</p>



<a name="217510810"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/source-based-code-coverage/near/217510810" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Diggsey <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/source-based-code-coverage.html#217510810">(Nov 21 2020 at 17:37)</a>:</h4>
<p>Has anyone managed to get source-based code coverage working with <code>codecov.io</code>?</p>



<a name="217617940"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/source-based-code-coverage/near/217617940" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Ben Chand <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/source-based-code-coverage.html#217617940">(Nov 23 2020 at 11:43)</a>:</h4>
<p>I'm a bit confused on how source based code coverage works for tests. Following the guide here:<br>
<a href="https://doc.rust-lang.org/nightly/unstable-book/compiler-flags/source-based-code-coverage.html">https://doc.rust-lang.org/nightly/unstable-book/compiler-flags/source-based-code-coverage.html</a></p>
<p>I don't particularly want to see the code coverage from running the binary itself, but from running the tests. I've tried running test binaries to generate profraw files with <code>RUSTFLAGS="-Zinstrument-coverage" cargo test</code> but the final report indicates that no lines have been hit.</p>
<p>Has anyone managed to get this working?</p>



<a name="217629054"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/source-based-code-coverage/near/217629054" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eric Huss <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/source-based-code-coverage.html#217629054">(Nov 23 2020 at 13:46)</a>:</h4>
<p><span class="user-mention" data-user-id="366687">@Ben Chand</span> You'll probably want to run it for one test at a time (since each one is a separate binary).  Here's an example:</p>
<div class="codehilite"><pre><span></span><code>RUSTFLAGS=&quot;-Zinstrument-coverage&quot; cargo +nightly test --lib
llvm-profdata merge -sparse *.profraw -o foo.profdata
llvm-cov show -Xdemangler=rustfilt target/debug/deps/foo-927f18c984bb5e6b -instr-profile=foo.profdata --show-line-counts-or-regions --output-dir=cov --format=html
open cov/index.html
</code></pre></div>
<p>The path to <code>llvm-cov</code> should be the test executable (which is printed with <code>cargo test</code>).</p>
<p>I'm not sure if it is possible to merge the results across multiple test executables.  You could maybe experiment with the <code>LLVM_PROFILE_FILE</code> environment variable with the <code>%m</code> variable as shown in <a href="#narrow/stream/246057-t-cargo/topic/meeting/near/216405039">https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo/topic/meeting/near/216405039</a>.</p>



<a name="217629710"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/source-based-code-coverage/near/217629710" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/source-based-code-coverage.html#217629710">(Nov 23 2020 at 13:52)</a>:</h4>
<p><span class="user-mention" data-user-id="120518">@Eric Huss</span> can we document that somewhere, maybe on the blog post? It looks super helpful :)</p>



<a name="217629934"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/source-based-code-coverage/near/217629934" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eric Huss <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/source-based-code-coverage.html#217629934">(Nov 23 2020 at 13:55)</a>:</h4>
<p>More examples and documentation in <a href="https://doc.rust-lang.org/nightly/unstable-book/compiler-flags/source-based-code-coverage.html">https://doc.rust-lang.org/nightly/unstable-book/compiler-flags/source-based-code-coverage.html</a> would be great.  I am no expert here, I'm making wild guesses on how to use it based on that document, so I wouldn't be comfortable writing it.</p>



<a name="217630207"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/source-based-code-coverage/near/217630207" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Ben Chand <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/source-based-code-coverage.html#217630207">(Nov 23 2020 at 13:57)</a>:</h4>
<p><span class="user-mention" data-user-id="120518">@Eric Huss</span> Thanks for that example, it works very well for me :) Great to see a working example</p>



<a name="217653556"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/source-based-code-coverage/near/217653556" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/source-based-code-coverage.html#217653556">(Nov 23 2020 at 16:55)</a>:</h4>
<p>cc <span class="user-mention" data-user-id="296355">@Rich Kadel</span></p>



<a name="217654377"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/source-based-code-coverage/near/217654377" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Ben Chand <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/source-based-code-coverage.html#217654377">(Nov 23 2020 at 17:01)</a>:</h4>
<p>So far I've managed to get test coverage working for some small local crates and it works really well. But I'm struggling with generating the profile data for one of our larger crates. Attempting to show the instr-profile results in the error <code>Failed to load coverage: Truncated coverage data</code>. Has anyone seen this before?</p>
<p>I've tried a number of different configurations and execution methods, but the most recent one which also failed is based on the approach here, where all *.profraw files are merged:<br>
<a href="#narrow/stream/246057-t-cargo/topic/meeting/near/216405039">https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo/topic/meeting/near/216405039</a></p>



<a name="217658443"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/source-based-code-coverage/near/217658443" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Rich Kadel <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/source-based-code-coverage.html#217658443">(Nov 23 2020 at 17:33)</a>:</h4>
<p>I have some updated instructions that I'll put in the <code>Unstable Book</code> soon. Specifically, I worked through a set of steps to simplify and (hopefully) clarify coverage for tests. FYI, yes, <code>%m</code> works and is a good idea if testing multiple binaries. Then you can merge the multiple <code>.profraw</code> results (from each test binary) with <code>llvm-profdata</code>, and you can run reports from the merged results by passing the path to all tested binaries to <code>llvm-cov</code>.</p>



<a name="217658503"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/source-based-code-coverage/near/217658503" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Rich Kadel <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/source-based-code-coverage.html#217658503">(Nov 23 2020 at 17:34)</a>:</h4>
<p>Here's a snapshot of the steps:</p>



<a name="217659809"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/source-based-code-coverage/near/217659809" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Rich Kadel <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/source-based-code-coverage.html#217659809">(Nov 23 2020 at 17:46)</a>:</h4>
<div class="codehilite" data-code-language="Bash"><pre><span></span><code>$ rustup toolchain install nightly
$ rustup default nightly
$ rustup update  <span class="c1"># some redundancy here, depending on whether you are already on nightly</span>
$ rustup component add llvm-tools-preview
$ cargo install cargo-binutils  <span class="c1"># this makes finding/using the right LLVM tool versions easier</span>
$ cargo install rustfilt
</code></pre></div>
<p>For the rest of the commands in the following example, I downloaded <code>json5format</code>. You can validate the commands with this, or skip it and choose your own crates of course:</p>
<div class="codehilite" data-code-language="Bash"><pre><span></span><code>$ git clone git@github.com:google/json5format.git
$ <span class="nb">cd</span> json5format
</code></pre></div>
<p>That completes all of the one-time setup for the following steps:</p>
<div class="codehilite" data-code-language="Bash"><pre><span></span><code>$ <span class="nv">RUSTFLAGS</span><span class="o">=</span><span class="s2">"-Z instrument-coverage"</span> <span class="se">\</span>
    <span class="nv">LLVM_PROFILE_FILE</span><span class="o">=</span><span class="s2">"json5format-%m.profraw"</span> <span class="se">\</span>
    cargo <span class="nb">test</span> --tests
</code></pre></div>
<p>Make note of the binary paths, after <code>Running</code>:</p>
<div class="codehilite" data-code-language="Text only"><pre><span></span><code>   ...
   Compiling json5format v0.1.3 (/usr/local/google/home/richkadel/json5format)
    Finished test [unoptimized + debuginfo] target(s) in 14.60s
     Running target/debug/deps/json5format-fececd4653271682
running 25 tests
...
test result: ok. 25 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

     Running target/debug/deps/lib-30768f9c53506dc5
running 31 tests
...
test result: ok. 31 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
</code></pre></div>
<p>You should have one ore more <code>.profraw</code> files now, one for each test binary.</p>
<p>Now run <code>profdata</code> once to merge them, and then run <code>cov</code> with the <code>profdata</code> file and all test binaries. Here I'm running both <code>cargo cov report</code> and <code>cargo cov show</code>. (<code>cargo profdata</code> wraps <code>llvm-profdata</code> and <code>cargo cov</code> wraps <code>llvm-cov</code>, but you need to prefix tool args with <code>--</code>. Or you can use <code>rust-profdata</code> and <code>rust-cov</code>, I believe, without <code>cargo</code>, and without the extra <code>--</code>.)</p>
<div class="codehilite" data-code-language="Bash"><pre><span></span><code>$ cargo profdata -- merge <span class="se">\</span>
    -sparse json5format-*.profraw -o json5format.profdata
$ cargo cov -- report <span class="se">\</span>
    --use-color --ignore-filename-regex<span class="o">=</span><span class="s1">'/usr/local/'</span> <span class="se">\</span>
    --instr-profile<span class="o">=</span>json5format.profdata <span class="se">\</span>
    target/debug/deps/lib-30768f9c53506dc5 <span class="se">\</span>
    target/debug/deps/json5format-fececd4653271682
$ cargo cov -- show <span class="se">\</span>
    --use-color --ignore-filename-regex<span class="o">=</span><span class="s1">'/usr/local/'</span> <span class="se">\</span>
    --instr-profile<span class="o">=</span>json5format.profdata <span class="se">\</span>
    target/debug/deps/lib-30768f9c53506dc5 <span class="se">\</span>
    target/debug/deps/json5format-fececd4653271682 <span class="se">\</span>
    --show-instantiations --show-line-counts-or-regions <span class="se">\</span>
    --Xdemangler<span class="o">=</span>rustfilt <span class="p">|</span> less -R
</code></pre></div>



<a name="217663057"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/source-based-code-coverage/near/217663057" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Rich Kadel <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/source-based-code-coverage.html#217663057">(Nov 23 2020 at 18:11)</a>:</h4>
<p>One thing that might be worth clarifying: the <code>cargo cov</code> argument <code>--ignore-filename-regex='/usr/local/' \</code>.</p>
<p>The steps above generate coverage results for everything in <code>json5format</code> <strong>plus</strong> all of the dependencies. In my case, the dependencies are loaded from an absolute path starting at <code>/usr/local</code>, and all of the <code>json5format</code> source files are loaded from relative paths.</p>
<p>I added that argument to skip reviewing coverage for the dependencies.</p>
<p>You can build the dependencies without coverage instrumentation, and then build the json5format code with coverage instrumentation. I haven't tried that in a while though, and I'm not sure if there are any undesired side-effects, like perhaps (<em>maybe but unconfirmed</em>) inlined functions and/or generics from dependencies might not be included, which may or may not be what you want or expect.</p>
<p>Something to figure out, because if the results are reasonable, it would save some time and binary size to exclude the dependencies from coverage instrumentation at compile time.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>